Estimation of lens distortion correction from single images

In this paper, we propose a method for estimation of camera lens distortion correction from a single image. Without relying on image EXIF, the method estimates the parameters of the correction by searching for a maximum energy of the so-called linear pattern introduced into the image during image acquisition prior to lens distortion correction. Potential applications of this technology include camera identification using sensor fingerprint, narrowing down the camera model, estimating the distance between the photographer and the subject, forgery detection, and improving the reliability of image steganalysis (detection of hidden data). 1. MOTIVATION Thanks to the increasing power of processors currently used in consumer digital cameras, manufacturers began implementing various computational imaging technology inside the camera to allow the photographer to take better images using cheaper hardware and optics. Most cameras today compensate for the imperfections of the optical system by correcting for the geometrical lens distortion (LD) and chromatic aberration inside the camera before saving the resulting image to the memory card. Such processing leaves detectable traces that can be exploited for a variety of forensic purposes. In this paper, we introduce a technique that can estimate the parameters of the lens distortion correction from a single image. The method makes use of the so-called linear pattern (LP) 2 commonly present in digital images. It manifests through non-zero sums of rows and columns in the image noise residual. The LP is caused by several processes that are applied at different times in the image acquisition pipeline. The first component of the LP, L0, is due to sensor signal readout; this part is present in the raw sensor output. Another component, L1, is added during color interpolation (demosaicking) and JPEG compression. The part of the LP that is introduced before the LD correction is applied becomes deformed in the image corrected for the LD and thus can be used as a template to estimate the parameters of the LD. The method proposed here estimates the parameters of the inverse LD correction transformation using the energy of the LP, particularly of the component L0. We work under the assumption that a forensic analyst has one image under investigation (the test image) and, optionally, knows the model of the camera that took the image. In particular, we assume no knowledge of the image EXIF header as this meta data can be replaced or lost when editing the image. Additionally, we assume that the image was not off-center cropped, rotated, or corrected for perspective distortion. The analyst desires to estimate the amount of LD correction that was applied to the test image. There are numerous potential applications of such an estimation method: 1. It can be used as a front-end for camera identification using sensor fingerprints. Sensor fingerprints must be resynchronized with the noise residual of the test image prior to applying a correlation-based detector. The proposed technique can be used to resynchronize the residual with the fingerprint, thus saving on processing time when a large number of fingerprints need to be checked. 2. By detecting the presence or absence of camera LD correction, one could potentially narrow down the model of the camera that took the test image, e.g., eliminate cameras whose optical system cannot not introduce distortion of a certain strength. The method could be potentially extended to these cases at the expense of increased computational complexity and lowered accuracy. 3. When the camera model is known, the estimated parameters could be used to determine the focal length (zoom) at which the image was taken. This could be further used to estimate the proximity of the photographer to the object. 4. The image resampling due to LD correction introduces a specific global structure of pixel correlations into the image that might be locally disturbed when an object in the image (region of interest) is replaced, deleted, or manipulated. This may be useful for forgery detection. 5. The symmetry of the deformed LP can also be used to determine the optical center of the image and thus detect off-center image cropping. This would require an appropriate extension of the LD model used in this paper. 6. It has been established that the reliability of steganalysis (detection of the presence of data hidden in an image) can be highly sensitive to local correlations within pixels or other resampling artifacts. Since LD correction involves resampling, knowing that an image, which is suspected to contain secretly embedded data, has been corrected for LD is important to the steganalyst. For example, the steganalyst can train a steganography detector on an appropriate cover source (images corrected for LD) to avoid the negative impact of the cover source mismatch 5, 9, 10 and improve the reliability of steganalysis. In the next section, we describe the lens distortion model that will be used in this paper and we also contrast this contribution with previous art. In Section 3, we define the concept of the linear pattern, describe the method for estimating the parameters of the lens distortion, and investigate further possible adjustments and their effect on parameter estimation accuracy. We also describe the method using which we determine the ground truth to allow for a quantitative assessment of the proposed method. All experimental results appear in Section 4. We experiment with a total of six cameras and report the percentage of successful determination of the lens distortion parameters including the cases when no distortion correction was applied. The paper is summarized in Section 5. 1.1 Notation and preliminaries Everywhere in this paper, boldface font will denote vectors (or matrices) of length specified in the text. For X and Y two m× n matrices (or m× n dimensional vectors), the Euclidean dot product is denoted as X · Y with ‖X‖ = √ X · X being the L2 (Euclidean) norm of X. Denoting the sample mean with a bar, the normalized correlation and the Peak Correlation to Energy (PCE) ratio are defined as ρ(X,Y) = (X − X) · (Y − Y) ∥X − X ∥∥Y − Y ∥∥ , PCE(X,Y) = ρ 1 mn−|N| ∑ s∈I−N ρ 2(s) , (1) where I = {1, . . . ,m} × {1, . . . , n} and N = {s = (s1, s2); (s1 ≤ 5 ∨ s1 ≥ m− 5) ∧ (s2 ≤ 5 ∨ s2 ≥ n− 5)} is a small neighborhood of s = (0, 0). For brevity, we use ρ(s) = ρ(xi+s1,j+s2 , yi,j) with the remark that the indices wrap up cyclically whenever they get out of their original ranges. 2. LENS DISTORTION MODEL A model of geometrical LD expresses how the lens distortion changes the original distance of each pixel to the image center, r: r = Ta(r) = r(1 + a2r 2 + a4r ), a = (a2, a4). (2) This model is commonly used for modeling barrel and pincushion optical distortion. The parameters a2 and a4 control the amount and type of the distortion. Our goal is to estimate one or both of these parameters from a single image. Notice that this model reflects our assumption that the image under investigation has not been cropped off-center as it assumes the optical center to be located exactly in the image geometrical center. This structure can be captured using e.g., the so-called p-map. We also point out that our goal is not to implement the best LD correction. The goal is to estimate LD correction that the manufacturer implemented in the camera. A previously published related study justifies this particular form of the model (2) for applications in digital image forensics. 2.1 Relationship to prior art We would like to make a clear distinction between the method proposed here and the prior art, which focused on extending the camera identification algorithm based on sensor fingerprint to images corrected for LD. The method employed in Ref. [4] used the sensor fingerprint to estimate the distortion parameters. In this paper, we cannot use the sensor fingerprint because all the analyst has is a single image. Instead, we make use of the part of the LP that was inserted into the test image prior to the LD correction. We also contrast the proposed work with the content of Section 6.2 from Ref. [4], where the parameter a2 was estimated using the energy of the LP. There, a high quality fingerprint was needed for the method to work reliably. Moreover, this cited work did not consider the two-parameter model (2), which is essential for the proposed method to work reliably for single images. 3. ESTIMATING LD PARAMETERS Our goal is to estimate the LD parameters a2 and a4 in (2) from a single image, when it is an unmodified JPEG output from a digital camera. We do so by repeatedly applying the inverse of the LD correction (2) with varying parameters to the image noise residual and detecting the largest local peak in its LP energy. The assumption we adopt is that the energy of the original LP is reduced by LD correction. Generally, the stronger the correction is, the lower the LP energy becomes. The proposed parameter estimation method starts with extracting the image noise component (also called the noise residual) from each color channel. Given an m × n 8-bit color channel, e.g., the red channel R ∈ {0, . . . , 255}m×n, its noise residual is obtained by subtracting from it its denoised version, W(R) = R − F (R), where F is a denoising filter. We use the filter described in Ref. [12], which employs Wiener filtering of wavelet coefficients in a Daubechies 8-tap wavelet transform (also see Appendix A of Ref. [11]). The noise residuals of all three R, G, B channels are merged to one m×n matrix W using the linear combination used for conversion from RGB to grayscale: W = 0.2126× W(R) + 0.7152× W(G) + 0.0722× W(B). (3) The LD parameter estimation method is based on the assumption that the energy of the linear pattern of W is maximized when the inverse of the LD correction transform is applied to it, T a (W). Before describing the individual steps of the algorithm in detail, we provide a formal definition of the linear pattern and its energy. 3.1 Linear pattern For a zero-mean matrix X, X = [xi,j ] m,n i,j=1, ∑ i,j xi,j = 0, we define the linear pattern of X as an ordered set L = L(X) = {c, r} of column averages c = (c1, c2, ..., cn) and row averages r = (r1, r2, ..., rm) of X: